Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SF-3162 Warn user when training and drafting sources are different #2982

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

RaymondLuong3
Copy link
Collaborator

@RaymondLuong3 RaymondLuong3 commented Jan 28, 2025

This PR adds a warning for users when their training and drafting sources have different languages. It is not so simple just to compare language codes because some languages may have multiple valid codes. My approach was to take the Intl.DisplayNames() function and compare the English names if the codes are different. I am open to suggestions for better approaches.

image


This change is Reviewable

@RaymondLuong3 RaymondLuong3 added the will require testing PR should not be merged until testers confirm testing is complete label Jan 28, 2025
Copy link

codecov bot commented Jan 28, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.04%. Comparing base (fd6844f) to head (8f68f24).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2982   +/-   ##
=======================================
  Coverage   82.04%   82.04%           
=======================================
  Files         544      544           
  Lines       31697    31712   +15     
  Branches     5155     5132   -23     
=======================================
+ Hits        26005    26018   +13     
- Misses       4925     4938   +13     
+ Partials      767      756   -11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@kylebuss kylebuss self-assigned this Jan 29, 2025
Copy link
Collaborator

@kylebuss kylebuss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this! Overall it looks good. I like the languageCodesEquivalent solution, it's simple and should cover the majority if not all cases we would see.

Rather than showing an additional warning message, what if we only show the "All languages are correct" notice if we determine there is a mismatch between training and drafting source languages? We could change the required languagesVerified message to "I understand, proceed drafting with selected languages".

Reviewed 9 of 9 files at r1, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @RaymondLuong3)


src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/confirm-sources/confirm-sources.component.ts line 88 at r1 (raw file):

  private sourceLanguagesAreCompatible(source: TranslateSource, other: TranslateSource): boolean {
    if (source.writingSystem.tag === other.writingSystem.tag) return true;
    return this.i18nService.languageCodesEquivalent(source.writingSystem.tag, other.writingSystem.tag);

nit I wonder if this should take a source[] that we loop through and compare source[0] to each item? Doing this now should prevent us from having to update if/when we go to more than 1 alternate source.

Code quote:

  private sourceLanguagesAreCompatible(source: TranslateSource, other: TranslateSource): boolean {
    if (source.writingSystem.tag === other.writingSystem.tag) return true;
    return this.i18nService.languageCodesEquivalent(source.writingSystem.tag, other.writingSystem.tag);

Copy link
Collaborator

@josephmyers josephmyers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the "language confirmation" notice, the existing direction is to have the notice showing all the time with the hopes that the user checks and confirms the languages every time, as a safeguard. @Nateowami may have more to say on this.

Reviewed all commit messages.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @RaymondLuong3)


src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/confirm-sources/confirm-sources.component.html line 55 at r1 (raw file):

  @if (showSourceLanguagesNotCompatibleError) {
    <app-notice type="error">
      <span>{{ t("sources_must_be_same_language") }}</span>

If there are multiple training sources and their languages differ, we show a message on the main drafting page and prevent them from even getting to the first step. I believe we should follow this pattern, unless there's a good reason not to.

image.png

Copy link
Collaborator

@kylebuss kylebuss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this PR changes that requirement. We've confirmed with the Serval/EITL teams that if we verify the language codes are equivalent then draft quality will not be degraded.

I guess the question on keeping it as a safeguard for all instances would be if we believe selected sources have a high probability of having an incorrect language code set that needs reviewed before proceeding? Otherwise, I'd suggest that only displaying it when we detect a different language code alerts the user draft quality would be reduced but still allow them the option to proceed.

Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @josephmyers and @RaymondLuong3)


src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/confirm-sources/confirm-sources.component.html line 55 at r1 (raw file):

Previously, josephmyers wrote…

If there are multiple training sources and their languages differ, we show a message on the main drafting page and prevent them from even getting to the first step. I believe we should follow this pattern, unless there's a good reason not to.

image.png

This is a good point. I would consider multiple training sources with different language codes a hard stop as I don't believe Serval can handle it. I could go either way on the changes in the PR being a hard or soft stop. Serval will handle training and drafting sources being different language codes (but draft quality will be poor).

Copy link
Collaborator

@josephmyers josephmyers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, cool. It sounds like you have more recent information than I do. Carry on!

Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @kylebuss and @RaymondLuong3)


src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/confirm-sources/confirm-sources.component.html line 55 at r1 (raw file):

Previously, kylebuss (Kyle Buss) wrote…

This is a good point. I would consider multiple training sources with different language codes a hard stop as I don't believe Serval can handle it. I could go either way on the changes in the PR being a hard or soft stop. Serval will handle training and drafting sources being different language codes (but draft quality will be poor).

Right. As the PR is currently, it is a hard stop, but it's hard stopping too late in the process. If we're going to hard stop, we need to hard stop at the "landing" page, where the other hard stops are.

@jellomoosie
Copy link

jellomoosie commented Feb 5, 2025

Given that the risk/cost of a poor-quality draft is not significant and can be resolved by the user, we should default toward warning but not prohibiting users from choosing settings that look like they are incorrect.

To clarify, the trade-off I'm assuming is either

  1. Warn users but allow them to proceed with some friction >> potential bad outcome = wasted time and compute resources to create a draft that's poor quality
  2. Warn users and prevent them from proceeding >> potential bad outcome = users cannot create a draft at all, or are forced to modify their language codes to something that may not make sense to them but complies with our comparison logic

The first bad outcome seems preferrable.

@RaymondLuong3
Copy link
Collaborator Author

Thanks for the considerations. I think the best way forward is to warn users on the confirm page when language codes are not equivalent between training and drafting languages, and then that means we do not have warnings on the generation page when their language codes do not match. The user will be able to proceed to generate the draft even if these warnings are visible

Copy link
Collaborator

@josephmyers josephmyers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this makes sense. Can you update the JIRA task (title, description, acceptance tests, etc.), as well as updating this PR, to match this new direction?

Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @kylebuss and @RaymondLuong3)

@RaymondLuong3 RaymondLuong3 force-pushed the improvement/sf-3162-language-code-confirm branch from 8f68f24 to 17d5624 Compare February 6, 2025 19:34
Copy link
Collaborator

@kylebuss kylebuss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am getting an error when starting a new draft.

ConsoleMessage.png
ErrorScreen.png

Reviewable status: 3 of 12 files reviewed, 2 unresolved discussions (waiting on @RaymondLuong3)

@RaymondLuong3 RaymondLuong3 marked this pull request as draft February 6, 2025 20:46
@RaymondLuong3
Copy link
Collaborator Author

Changing to be a draft

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
will require testing PR should not be merged until testers confirm testing is complete
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants